Detecting objects within a vehicle in connection with a service

ABSTRACT

A system for monitoring a vehicle used in providing a service is disclosed. The system detects when the service is complete, and upon detecting that the service is complete, the system determines, from one or more sensors of the vehicle, that an object which should have been removed from the vehicle after completion of the transport service, remains in the vehicle. The system then automatically initiates performance of a remedial action.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/629,372, filed Jun. 21, 2017, titled “DETECTING OBJECTS WITHIN AVEHICLE IN CONNECTION WITH A SERVICE”, which is a continuation of U.S.patent application Ser. No. 15/449,110, filed Mar. 3, 2017, now U.S.Pat. No. 9,790,729 issued Jun. 22, 2017, titled “DETECTING OBJECTSWITHIN A VEHICLE IN CONNECTION WITH A SERVICE”, which is a continuationof U.S. patent application Ser. No. 14/708,611 filed May 11, 2015, nowU.S. Pat. No. 9,616,773 issued Apr. 11, 2017, titled “DETECTING OBJECTSWITHIN A VEHICLE IN CONNECTION WITH A SERVICE”, the aforementionedpriority applications being hereby incorporated by reference herein forall purposes.

BACKGROUND

A service arrangement system can arrange a service to be provided for arequesting user through the use of mobile computing devices. Forexample, a user can request a service, such as a transport or deliveryservice, by operating the user's mobile computing device, and theservice arrangement system can process the request to select theappropriate provider that can provide the service. The servicearrangement system can also monitor the service, such as to determinethe current location of the vehicle, before and during the progress ofthe service.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example system to detect objects within a vehiclein connection with a transport service.

FIG. 2 illustrates an example method for detecting objects within avehicle in connection with a transport service.

FIG. 3 illustrates another example method for detecting objects within avehicle in connection with a transport service.

FIGS. 4A through 4D illustrate examples of user interfaces displayed ona mobile computing device in various use case examples.

FIG. 5 is a block diagram that illustrates a computing device upon whichembodiments described herein may be implemented.

FIG. 6 is a block diagram that illustrates a computer system upon whichexamples described herein may be implemented.

DETAILED DESCRIPTION

According to examples described herein, a service arrangement system,which implement a network service, can receive requests forlocation-based services and arrange those services to be provided byservice providers. As used herein, service providers can include humandrivers (or human-driven vehicles) that receive and/or acceptinvitations for services, and in some instances, autonomous (orself-driving) vehicles that can be used by the network service totransport goods or people. Whether a human-driven vehicle or anautonomous vehicle (AV) is assigned to provide a service, the networkservice can use information from a set of sensors of that vehicle toperform processes, when necessary, to supplement the assigned service.For example, a computing system associated with a vehicle can determine,through use of sensor data, whether the service has been properlycompleted or whether an object has inadvertently been left behind in thevehicle.

In some examples, a computing system can determine that a vehicle hasbeen arranged to provide a location-based service. Depending onimplementation, the computing system can correspond to a computingsystem associated with a vehicle (e.g., an on-board computing system ofthe vehicle, or a driver device operated by the driver of the vehicle)or a computing system(s) that implements the network service (e.g., abackend server system remote from the vehicle). During the performanceof the service, the computing system can detect that an object has beenplaced within the vehicle based on a first set of information from oneor more sensors of the vehicle at a first time. As described herein, asensor is a device that can detect some characteristic of thesurrounding environment or detect an event or change in quantity of anamount, and provide a corresponding output (e.g., sensor data). Thecomputing system can detect, at the time the service is being completedor after the service has been completed, whether the object has beenremoved from the vehicle based on a second set of information from theone or more sensors. In response to detecting that the object is stillpresent in the vehicle (e.g., has not been removed from the vehicle),the computing system can automatically perform a remedial action, suchas causing the driver or the vehicle to perform an additional operationthat would otherwise not have been performed.

In various examples, a vehicle can include or be equipped with one ormore sensors, such as a set of cameras, a set of weighing devices, a setof suspension sensors, a set of light detection and ranging (LIDAR)sensors, a set of ultrasound sensors, a set of location-determinationsensors, a set of radio-frequency sensors (such as Bluetooth or Wi-Fitransceivers), as well as other sensors used for various vehicleoperations (e.g., engine sensors, tire pressure sensors, door open orclose sensors, seat belt sensors, external stereo cameras or LIDARsensors for AVs, etc.). A set of sensors, as described herein, cancorrespond to one or more individual sensors or one or more arrays ofsensors. Depending on implementation, the computing system can useinformation (sensor data) from one or more sets of sensors (e.g., anindividual sensor or a combination of a plurality of different sensors)to detect whether an object has been placed within the vehicle and/orhas been removed from the vehicle.

For example, the computing system can detect a position of an objectinside the vehicle based on image information from a set of cameras thatare positioned within the vehicle with the respective lens of the set ofcameras facing towards the interior of the vehicle. In another example,the computing system can detect that an object has been placed in orremoved from a compartment or the interior of the vehicle based onweight information from a set of weighing devices that are positionedwithin the compartment or the interior of the vehicle. The set ofweighing devices can be positioned in the vehicle such that therespective measuring platform of the set of weighing devices ispositioned on, positioned below, or included with a bottom surface ofthe respective compartment or interior of the vehicle.

In another example, the computing system can detect that a mobilecomputing device with wireless communications has been placed in orremoved from the vehicle based on if that mobile computing device iscommunicating with antennas in the vehicle that use wireless protocolssuch as Bluetooth or Wi-Fi. In another example, if the mobile computingdevice has its own location sensors, the computing system can comparethe output of those location sensors to the output of location sensorson the vehicle to determine if the mobile computing device is movingwith the vehicle, and is therefore likely to be inside the vehicle.Still further, in another example, the computing system can detect thatan object has been placed in or removed from the compartment or theinterior of the vehicle based on compression information from a set ofsuspension sensors that are positioned proximate to or included in asuspension system of the vehicle.

If the computing system detects that an object has been placed withinthe vehicle as part of or during the performance of the on-demandservice, and subsequently detects, at the time the service is beingcompleted or after the service has been completed, that the object isstill present in the vehicle, the computing system can automaticallyperform a remedial action, such as communicating a message to acomputing device associated with the user that requested the serviceand/or notifying the service arrangement system that the object is stillpresent in the vehicle. According to some examples, the computing systemcan also instruct the driver or cause the vehicle to remain stationaryat a location of the vehicle when the object was detected to still bepresent in the vehicle for at least a predetermined duration of time. Inanother example, the computing system can instruct the driver or causethe vehicle to travel to a predetermined or specific location where theobject can be removed from the vehicle.

According to an example, the computing system can also use sensor datato identify what the object is or what type of object has been placed inthe vehicle. Still further, in some examples, the computing system canalso use the detection of an object being placed in the vehicle and/orthe detection of the object being removed from the vehicle to determine(or as part of determining) when the service has started and/or ended.

As used herein, a client device, a driver device, and/or a mobilecomputing device refer to devices corresponding to desktop computers,cellular devices or smartphones, personal digital assistants (PDAs),laptop computers, tablet devices, etc., that can provide networkconnectivity and processing resources for communicating with a remotecomputing system(s) over one or more networks, such as a servicearrangement system. A remote computing system can refer to one or morecomputing systems or servers that is remote from the client device, thedriver device, and/or the vehicle (e.g., corresponds to the back-endserver system of the network service). Still further, in examplesdescribed herein, a driver device or a computing system associated witha vehicle can also correspond to custom hardware of the vehicle, such asan in-vehicle or on-board computing system, that has networkconnectivity and location-determination capabilities.

Still further, examples described herein relate to a variety oflocation-based (and/or on-demand) services, such as a transport service,a food truck service, a delivery service, an entertainment service, etc.to be arranged between users and service providers. In other examples, aservice arrangement system can be implemented by any entity thatprovides goods or services for purchase through the use of computingdevices and network(s). For purpose of simplicity, in examples describedherein, the service arrangement system can correspond to a transportarrangement system that arranges transport services to be provided forriders/users by drivers or vehicles (e.g., transporting objects orpeople).

One or more examples described herein provide that methods, techniques,and actions performed by a computing device are performedprogrammatically, or as a computer-implemented method. Programmatically,as used herein, means through the use of code or computer-executableinstructions. These instructions can be stored in one or more memoryresources of the computing device. A programmatically performed step mayor may not be automatic.

One or more examples described herein can be implemented usingprogrammatic modules, engines, or components. A programmatic module,engine, or component can include a program, a sub-routine, a portion ofa program, or a software component or a hardware component capable ofperforming one or more stated tasks or functions. As used herein, amodule or component can exist on a hardware component independently ofother modules or components. Alternatively, a module or component can bea shared element or process of other modules, programs or machines.

Some examples described herein can generally require the use ofcomputing devices, including processing and memory resources. Forexample, one or more examples described herein may be implemented, inwhole or in part, on computing devices such as servers, desktopcomputers, cellular or smartphones, personal digital assistants (e.g.,PDAs), laptop computers, printers, digital picture frames, networkequipment (e.g., routers) and tablet devices. Memory, processing, andnetwork resources may all be used in connection with the establishment,use, or performance of any example described herein (including with theperformance of any method or with the implementation of any system).

Furthermore, one or more examples described herein may be implementedthrough the use of instructions that are executable by one or moreprocessors. These instructions may be carried on a computer-readablemedium. Machines shown or described with figures below provide examplesof processing resources and computer-readable mediums on whichinstructions for implementing examples described herein can be carriedand/or executed. In particular, the numerous machines shown withexamples described herein include processor(s) and various forms ofmemory for holding data and instructions. Examples of computer-readablemediums include permanent memory storage devices, such as hard drives onpersonal computers or servers. Other examples of computer storagemediums include portable storage units, such as CD or DVD units, flashmemory (such as carried on smartphones, multifunctional devices ortablets), and magnetic memory. Computers, terminals, network enableddevices (e.g., mobile devices, such as cell phones) are all examples ofmachines and devices that utilize processors, memory, and instructionsstored on computer-readable mediums. Additionally, examples may beimplemented in the form of computer-programs, or a computer usablecarrier medium capable of carrying such a program.

System Description

FIG. 1 illustrates an example system to detect objects within a vehiclein connection with a transport service. Depending on implementation,such a system can be implemented by and operated on a mobile computingdevice of a driver, an in-vehicle computing system of a vehicle, or aservice arrangement system that provides a network service. For purposeof simplicity, in the example of FIG. 1, the system 100 is described asbeing implemented by and operated on an in-vehicle computing system of avehicle, such as an AV, through execution of instructions stored in oneor more memory resources of the in-vehicle computing system. Asdescribed herein, the system 100 can include, be a part of, orcorrespond to an application or an operating system of the in-vehiclecomputing system that communicates with a service arrangement system 190over one or more networks.

According to examples, the system 100 can include a system manage 110, alocation determination 120, an object detection 130, a location database140, a trip database 150, and a service interface 160. For purpose ofsimplicity, other components of the system 100, such as a user interfacecomponent, other databases, device interfaces to communicate with devicecomponents, etc., are not illustrated in FIG. 1. The components of thesystem 100 can combine to detect objects that are placed within and/orremoved from a vehicle in connection with transport services. Logic canbe implemented with various applications (e.g., software) and/or withhardware of the computing system that implements the system 100.

In one example, the system 100 can communicate with the servicearrangement system 190 via the service interface 160. As describedherein, the service arrangement system 190 can communicate with aplurality of service providers (e.g., driver devices and/or in-vehiclecomputing systems) and a plurality of client (or user) devices in orderto arrange transport services between users and service providers. Theservice arrangement system 190 can be implemented on network sideresources, such as on one or more servers or data centers, orimplemented through other computer systems in alternative architectures(e.g., peer-to-peer networks, etc.). Typically, a user can operate aclient or user device (not shown in FIG. 1 for simplicity) tocommunicate with the service arrangement system 190 over a network(s)and make a request for a transport service at a specified location(e.g., a pickup location of the package or a person to be transported).In some examples, the user device can run a designated serviceapplication that communicates with the service arrangement system 190.The service arrangement system 190 can receive the request andprogrammatically select a service provider to provide the transportservice.

Depending on implementation, if the service arrangement system 190selects a human driver to provide the service, the service arrangementsystem 190 can transmit an invitation to the selected driver's device,thereby providing the driver with an option to accept or reject theinvitation to provide the transport service for the requesting rider. Inanother example, if the service arrangement system 190 selects an AVfrom a plurality of available AVs (and/or human drivers) to provide theservice, such as described in FIG. 1, the service arrangement system 190can transmit information about the service (referred to as serviceinstructions 191) to the system 100 via the service interface 160.

According to examples, the service interface 160 enables the system 100to exchange data between the system 100 and the service arrangementsystem 190. For example, the service interface 160 can use one or morenetwork resources of the in-vehicle computing system to exchangecommunications over one or more wireless networks (e.g., a cellulartransceiver, a WLAN transceiver, etc.). The service interface 160 caninclude or use an application programming interface (API), such as anexternally facing API, to communicate data with the service arrangementsystem 190. The externally facing API can provide access to the servicearrangement system 190 via secure access channels over the networkthrough any number of methods, such as web-based forms, programmaticaccess via restful APIs, Simple Object Access Protocol (SOAP), remoteprocedure call (RPC), scripting access, etc.

In the example of FIG. 1, the service arrangement system 190 hasprocessed a request for a transport service by a user and has selectedthe AV that implements the system 100 to provide the transport servicefor the requesting user. The service arrangement system 190 may havedetermined, based on the pickup location specified by the requestinguser and the current locations (or most recently determined locations)of available service providers nearby the pickup location (e.g., withina predetermined distance of the pickup location), that the AVimplementing the system 100 is to provide the transport service. Theservice arrangement system 190 can transmit service instructions 191about the transport service to instruct the vehicle about where to goand/or what to do. In some examples, the service instructions 191 caninclude user information 192 (e.g., a user identifier, contactinformation of the user, a user's rating, etc.), pickup location 193specified by the user (e.g., an address, a landmark, a location datapoint corresponding to a latitude and longitude coordinate), destinationlocation 194, and/or other information, such as the type of transportservice (e.g., transport a person, deliver a package, etc.) or detailsabout the number of people or number of packages. The serviceinstructions 191 can, in one example, cause the vehicle controllingcomponent(s) of the AV to control, at least in part, the AV to travel tothe appropriate locations. In other words, the vehicle controlcomponent(s) can use some or all of the service instructions 191 (inaddition to other information) to control the vehicle to move to thepickup location and/or the destination or drop off location.

The system manage 110 can receive the service instructions 191 and inresponse, can perform a plurality of operations on behalf of the system100. For example, the system manage 110 can process the serviceinstructions 191 by determining information about the transport serviceit is to perform, and can initiate the performance of the transportservice by determining the route(s) to travel to the pickup locationand/or the destination location and controlling the AV to travel to thepickup location and/or the destination location.

In one example, a trip manage component of the system manage 110 cancreate a trip entry 151 for the transport service using some or all ofthe information from the service instructions 191 and store the tripentry 151 in a trip database 150 of the system 100. The trip managecomponent can then update the trip entry 151 with information about thevehicle's location as the vehicle performs the transport service (e.g.,travels to the pickup location from the current location, picks up theuser or an object, travels to the drop off location, etc.). The tripmanage component can also record, in the trip entry 151, the time andthe location of the vehicle when the service instructions 191 werereceived, when the service started, and when the service ended. The tripentry 151 can be used by the system 100 and/or the service arrangementsystem 190 to determine the cost for the transport service for therequesting user (e.g., after completion) and for maintaining ahistorical record of the AV and/or the requesting user.

The location determination 120 can provide the system manage 110 withinformation about the vehicle's location by receiving location datapoints 121 (e.g., periodically) from one or more geo-aware resources ofthe in-vehicle computing system. For example, the location determination120 can periodically determine the current location of the vehicle bycommunicating with, e.g., via a device interface, the GPS receiver ofthe driver device, one or more other sensors, and/or one or moretransceivers (e.g., cellular transceiver, Wi-Fi transceiver) for purposeof determining the current location data point 121 of the vehicle. Inone example, the location determination 120 can make a call to the GPSreceiver periodically, such as every two seconds or four seconds, etc.,to periodically receive a current location data point 121 of the driverdevice. The location determination 120 can also determine the associatedtimestamp of each determined location data point 121 of the vehicle.Depending on implementation, the location determination 120 can receivea timestamp (and/or an error value) along with the location data point121 of the vehicle from the GPS receiver or can determine the timestampof the location data point 121 using a separate clock (e.g., an internalclock of the driver device). The system manage 110 can store thelocation data points 121 in the location database 140 and/or the tripentry 151 to maintain data about where the vehicle traveled inconnection with transport services. As an addition or an alternative,the location determination 120 can store the location data points 121 ofthe vehicle in the location database 140.

Referring back to the system manage 110, the trip manage component canalso use the location data points 121 from the location determination120 to monitor the vehicle. The trip manage component can determine theposition and the state of the vehicle or system 100 in connection withthe performance of the transport service (e.g., what stage of thetransport service the vehicle is at). For example, the state of thesystem 100 may previously have been in an available state, whichcorresponds to a state in which the vehicle can be assigned thetransport service from the service arrangement system 190. When thesystem 100 receives the service instructions 191, the trip managecomponent can place the system 100 in a different state, e.g., on routestate, which corresponds to the vehicle traveling to the pickuplocation. Similarly, each time the trip manage component determines thatthe transport service has initiated or ended, the trip manage componentcan place the system 100 in a respective state (e.g., change from the onroute state to the on trip state, or change from the on trip state tocompleted or available state, respectively). The trip manage componentcan store information about the state and/or when the state changeoccurred along with the corresponding location data points 121 in thetrip entry 151 of the transport service.

Still further, as an addition or an alternative, the service arrangementsystem 190 can locally generate and store its own trip entry thatcorresponds to or is similar to the trip entry 151 stored in the system100. The trip manage component can periodically provide theabove-described information about the transport service to the servicearrangement system 190, e.g., via the state information 111 and/or thelocation data points 113. The service arrangement system 190 can use thereceived information from the system 100 to provide updated informationabout the transport service to the requesting user (e.g., the locationof the AV and/or the current state of the AV, etc.). In addition, theservice arrangement system 190 can determine, from the state information111 and/or the location data points 113, when the transport service hasinitiated or completed and/or when the AV is available to provideanother transport service.

In addition to creating and updating a trip entry 151 for the transportservice, the system manage 110 can initiate the performance of thetransport service based, at least in part, on the service instructions191. According to some examples, a vehicle control component of thesystem manage 110 can control the AV to perform the transport service.Depending on implementation, the vehicle control component can interfacewith the AV control system or can correspond to the AV control system.As described herein, an AV includes (i) computing resources thatimplement the system 100 and/or the AV control system, (ii) othersub-components or sub-subsystems to control various other operations orparts of the AV, and (iii) a plurality of vehicle parts. For purpose ofsimplicity, not all parts of an AV are discussed herein.

According to examples, various parts of an AV can include, but are notlimited to, a frame or a chassis, a body (e.g., including doors,bumpers, windows, a cab, seats, a trunk space, etc.), tires, asuspension system, a braking system, a steering system, an electricalsystem (e.g., including batteries, wires, cables, alarm sub-system,etc.), a transmission system (e.g., including a drive train, engine,etc.), a fuel system, an exhaust system, a cooling system, and an AVcontrol system. The AV can also include electronic control units (ECUs),which include controllers and/or microcontrollers, and other electroniccomponents that can control the various systems or sub-systems of theAV, such as a door control unit, an engine control unit, a speed controlunit, a transmission control unit, etc. The AV control system can beprogrammed, through execution of instructions stored in memory by one ormore processors, to control the AV by sending control signals and byreceiving data (e.g., feedback or sensor data) from the various systemsand vehicle parts. For purposes of this application, the AV controlsystem can be instructed to control the AV so that it can safelynavigate roads and streets to move from one location to another inconnection with transport services.

Still further, the AV can include a location determination module orresources, such as a GPS receiver and/or wireless transceivers, todetermine the location of the AV relative to other objects or inreference to a coordinate system. The AV can also include a plurality ofsensors, such as a set of cameras, a set of stereo cameras, a set ofradars, a set of LIDAR sensors, etc. to enable the AV to detect thereal-time environment of the AV. One or more (or all) of these sensorscan be positioned on the exterior of the vehicle (e.g., on the front,rear, roof of the vehicle, etc.) to detect information about theenvironment around the AV, such as the position of the vehicle relativeto roads, building, other vehicles, etc., or the position of inanimateobjects. For example, in order for the AV to travel from a currentposition or location (e.g., determined from the GPS receiver as alocation data point 121) to a specified location (e.g., the pickuplocation 193 and/or destination location 194 indicated in the serviceinstructions 191), the AV control system can access a stored detailedspatial map(s) of the region it is in (e.g., including map data 112) anddetermine the route(s) to travel to the specified location. Based on thedetected environment surrounding the AV (from data periodically andcontinuously detected by the set of sensors), the AV control system canperiodically perform decision making operations that instruct the AV totake a certain action(s) to move to the specified location. As a resultof each decision making operation, the AV control system can determinethat it is to travel in a certain direction at a certain speed at oneinstance in time, for example, and at a next instance in time, candetermine that it is to decelerate to a certain speed, changebearings/directions slightly, or change lanes, etc. The AV controlsystem can provide control signals to the various vehiclecomponents/systems accordingly.

In one example, when the AV travels to the pickup location to initiatethe transport service, the system 100 can periodically determine whetherthe AV has reached the specified pickup location (e.g., start location).In other words, the system 100 can periodically check when the AVarrives at or is within a predetermined distance (or is within apredetermined estimated travel time) from the pickup location. Forexample, the trip manage component can periodically compare the currentlocation (e.g., the latest location data point 121 provided by thelocation determination 120) with the pickup location 193 and determinewhether the AV is within the predetermined distance or estimated traveltime. When the trip manage component determines that the AV is withinthe predetermined distance or estimated travel time, the trip managecomponent can change the state of the system 100 from the on route stateto the arriving/has arrived state. The state information 111 about thischange can be transmitted to the service arrangement system 190 so thatthe service arrangement system 190 can provide updated information aboutthe AV to the requesting user's device.

In addition, according to an example of FIG. 1, the system 100 can alsodetermine whether an object(s) has been placed in the vehicle inconnection with the transport service. In some instances, a driver or anAV control system may need to determine if a requesting user has placedan object in the vehicle while being transported (e.g., luggage, boxes,groceries) or determine if the object has been placed in the vehicle fora delivery transport service. As described herein, an object cancorrespond to an item(s) or a person(s) that is placed within or entersa vehicle. The object detection 130 can communicate with one or moresets of sensors or one or more combinations of sensors of the AV, viaone or more vehicle or sensor interfaces, to receive sensor data 131,and based on the sensor data 131, can determine whether an object hasbeen placed in (or removed from) the AV (referred to herein as an objectdetection process or operation). In one example, the object detection130 can transmit a set of control signals to the sensors to cause (ortrigger) the sensors to provide the sensor data 131. Such sensors caninclude a set of cameras, a set of weighing devices, a set of suspensionsensors, a set of light detection and ranging (LIDAR) sensors, a set ofradio-frequency antennas and/or transceivers, and/or a set of ultrasoundsensors that are positioned in or provided with the vehicle to detectthe presence of, the position of, and/or the weight of one or moreobjects.

In some examples, the object detection 130 can perform an objectdetection process by (i) receiving sensor data 131 from one or moresensors, (ii) processing the received sensor data 131 using otherinformation (e.g., models or previously stored data), and (iii)determining whether an object has been placed in the AV (when it waspreviously not in the AV) or determining whether an object has beenremoved from the AV (when it was previously in the AV). For example, theobject detection 130 can receive image information from one or morecameras positioned within a trunk of the AV (e.g., with the lens facingthe inside of the trunk), and process the image information by usingimage processing or image recognition processes and/or by comparing theimage information to previously stored baseline image information aboutthe trunk (e.g., showing the image of the trunk when it was empty or theimage of the trunk before receiving the service instructions 191). Thestored baseline image information can be a predefined image or can be animage captured by the one or more cameras at a time before the AVarrived at or near the pickup location 193. Still further, in oneexample, the object detection 130 can use image processing (as well asother stored or received data from the service arrangement system 190)to identify what the object is based on visual characteristics of theobject in the image.

In other examples, when one or more other sensors are used for detectingthe presence or position of an object, the object detection 130 can useother baseline or model information as a basis for comparison withreceived sensor data 131. For example, the object detection 130 canreceive sensor data 131 from one or more weighing devices or sensorsthat are positioned in a compartment of the AV such that its respectivemeasuring platform is on, positioned below, or included with a bottomsurface of such compartment. The object detection 130 can determine achange in the measurements from the sensor data 131 at a time before theAV arrived at or near the pickup location 193 and after the service hasstarted.

Depending on implementation, the object detection 130 can perform theobject detection process at various times. For example, the objectdetection 130 can concurrently perform the object detection process asthe AV travels to the pickup location 193 and/or as the trip managecomponent determines whether the AV is within the predetermined distanceor predetermined estimated travel time of the pickup location 193. Theobject detection 130 can make calls to the set of sensors to receivesensor data 131 periodically. In another example, the object detection130 can perform the object detection process at various instances wheninstructed or triggered by the system manage 110 (via trigger 115). Forexample, when the trip manage component determines that the AV is withinthe predetermined distance or predetermined estimated travel time of thepickup location 193, the trip manage component can transmit a trigger115 to cause the object detection 130 to perform the object detectionprocess. Still further, as another variation, the trip manage componentcan transmit the trigger 115 when it determines that the AV isstationary for a predetermined duration of time and/or when itdetermines that the AV is stationary for a predetermined duration oftime while it the AV is within the predetermined distance orpredetermined estimated travel time of the pickup location 193. Asanother example, the object detection 130 can be triggered to performthe object detection process when the trip manage component determinesthat the transport service has started. By causing the object detection130 to perform the object detection process at select times, the system100 can reduce excessive consumption of energy and computing resourcesby causing the sensor(s) to wake up (from a suspended or sleep state)and detect or perform a measurement only when necessary.

If the object detection 130 detects the position of an object or detectsthat an object has been placed within the vehicle based on a first setof sensor data 131, the object detection 130 can provide objectinformation 133 to the system manage 110. The object information 133 canindicate to the system manage 110 that an object has been placed in theAV (e.g., a message, or a binary bit can be toggled) and/or can includeinformation about that object based on the types of sensor(s) that isused by the AV to perform the detection. For example, the additionalinformation can include (i) what the object is (e.g., a box, a bag, askateboard, etc.), (ii) one or more images of the object captured by theset of cameras, if cameras are used, (iii) measurements pertaining tothe object (e.g., a weight, if weighing devices or suspension sensorsare used, or dimensions of the object, if cameras, LIDAR sensors orultrasound sensors are used, etc.), and/or (iv) where the object islocated (e.g., what compartment, or the position of the object). Thetrip manage component can store some or all of the object information133 with the trip entry 151 associated with the transport service, sothat information about the object can be logged and associated with theparticular user.

As described, concurrently (or before or after) the object detection 130detects an object, the trip manage component can determine that thetransport service has begun. In some examples, the system manage 110 candetermine that the transport service has started, based, at least inpart, on detecting that an object has been placed in the vehicle. Thetransport service can correspond to transporting a person (and/or theperson's things) from one location to another, or transporting an objectfor delivery from one location to another. The object detection 130 candetect, from one or more sensors, such as a camera or weighing devices,if a person and/or objects have entered the vehicle or if an object(without the person) has been placed in the vehicle. In this example,the set of cameras (or LIDAR sensors or ultrasound sensors) can bepositioned inside the cab/interior of the vehicle to enable the objectdetection 130 to detect that the person has entered the vehicle usingimage information from the camera(s) and processing the image(s). As anaddition or an alternative, a set of weighing devices or a set ofsuspension sensors that is positioned proximate to or included in asuspension system of the AV can determine that an amount of mass hasbeen added to the vehicle.

Once the transport service has started, the trip manage component canstore information about the performance of the transport service (e.g.,the time and/or the location of travel of the AV), including the starttime and start location of the transport service, in the trip entry 151.As the AV travels to the destination location 194 of the transportservice, the trip manage component can also use the location data points121 to determine when the AV approaches and has approached thedestination location 194, for purpose of determined, at least in part,that the transport service has completed.

In one example, in instances when the object detection 130 previouslydetected an object in connection with the transport service (e.g., therider placed a bag in the trunk of the AV), the trip manage componentcan cause the object detection 130 to also detect whether the object hasbeen removed from the vehicle. The object detection 130 can detectwhether the object has been removed from the vehicle at different times,depending on variations. According to one example, the system manage 110can cause the object detection 130 to concurrently perform the objectdetection process as the AV travels to the destination location 194(e.g., periodically), or perform the object detection process when thetrip management component determines that the AV is within apredetermined distance or predetermined estimated travel time of thedestination location 194. As an addition or an alternative, the tripmanage component can cause the object detection 130 to perform theobject detection process when the trip manage component determines thatthe AV is stationary for a predetermined duration of time, when the tripmanage component determines that the AV is stationary for apredetermined duration of time while it the AV is within thepredetermined distance or predetermined estimated travel time of thedestination location 194, or in response to determining that thetransport service has been completed.

When the trip manage component determines that the transport service hasbeen completed, for example, the object detection 130 can perform theobject detection process to determine if the object has been removedusing sensor data 131 received from the set of sensors (e.g., a secondset of sensor data 131). If the object detection 130 detects that theobject is still present, it can provide information to the system manage110 indicating that the object is still present in the vehicle. Inresponse, the system manage 110 can automatically perform a remedialaction. On the other hand, if the object detection 130 detects that theobject has been removed, no remedial process or action is taken. In onevariation, if the system 100 detects that the object has been removed(e.g., in the case of a delivery transport service), the system 100 candetermine that the delivery transport service has been completed.

As described herein, a remedial action can correspond to an operation(or set of operations) the system 100 performs to return whatever objectis accidentally left behind to the user or to make it easier for theuser to retrieve the object. In various implementations, the remedialaction(s) can be performed by the system 100 and/or performed by thesystem 100 in connection with the service arrangement system 190. In thelatter example, the remedial action(s) can be performed as a result ofthe exchange of data between the system 100 and the service arrangementsystem 190.

In one example, the system manage 110 can receive information that theobject is still remaining in the vehicle and/or information about theobject itself (e.g., based on recently determined information about theobject, or from object information 133 previously received when theobject was previously detected) and can generate an object message 117based on such information. The object message 117 can include, forexample, a time and/or a location when the transport service wascompleted, an identifier of the AV, an identifier of the user, anidentifier of the transport service, and information about the object(e.g., what the object is, one or more images of the object,measurements pertaining to the object, where the object is located). Thesystem manage 110 can determine a communication identifier for the user(e.g., a phone number or email address of the user or temporarilyassociated with the user's account), and transmit the object message 117to the user's computing device. Such an object message 117 can alsoinclude notification information to cause the user's computing device todisplay content to inform the user that the object has been left behindin the AV and/or to contact the entity that provides the servicearrangement system 190 (e.g., via the designated application running onthe user's computing device). As an addition or an alternative, anobject message 117 can be transmitted to the service arrangement system190, which can then transmit a message, including notificationinformation, to the user's computing device. The notificationinformation displayed to the user can provide information about theobject and/or how the user can retrieve the object. Examples ofdifferent notification information is illustrated in FIGS. 4A through4D.

As an addition or an alternative, the system manage 110 can also performone or more remedial actions by positioning the AV at a particularlocation (e.g., by causing the AV to move to a particular location orkeeping stationary at a particular location). For example, when aremedial action is to be performed, the system manage 110 can accessstored positioning instructions that instruct the vehicle controlcomponent to control, via the vehicle components, the AV to remainstationary at (i) the destination location 194, or (ii) the location ofthe vehicle when the object detection 130 determined that the object wasstill present in the vehicle. By causing the AV to remain stationary ata specified location, the object message 117 can inform the user thatthe user's object has been left behind in the AV, and that the AV willremain at the specified location for a predetermined duration of time(e.g., two minutes). Alternatively, the stored positioning instructionscan instruct the vehicle control component to cause the AV to travel toa specified location (e.g., a storage location or office building wherelost objects are stored), and the object message 117 can inform the userthat the object can be retrieved at the specified location duringcertain times.

In some examples where the AV travels to a specified location, such as astorage location or a disposal location, the AV can useautomatically-controlled actuators to remove the object from the AV forstorage or disposal. For example, the AV itself can include one or moreactuators coupled to one or more mechanical hands in the vehicle'sinterior or trunk that can grab an object and remove it. In anotherexample, the AV can move to a specified lane or spot at the storage ordisposal location, open the vehicle doors and/or trunk, and anautomatically-controlled mechanical arm and hand or claw can remove theobject from the vehicle.

Still further, in another example, the system manage 110 can provideinformation to the service arrangement system 190 indicating that anobject has been left behind, and in response, the service arrangementsystem 190 can transmit instructions to the system 100 to cause the AVto perform one or more tasks. In one implementation, the servicearrangement system 190 can transmit vehicle instructions to the systemmanage 110 that causes the vehicle control component to control themovement and operation of the AV. The vehicle instructions can cause theAV to move to a specified location or remain stationary at a specifiedlocation.

In some instances, after dropping off the user at the destinationlocation 194, the AV may be arranged/selected to provide a transportservice to another requesting user. As a result, the system 100 mayperform the remedial action, such as transmitting the object message117, but also cause the AV to travel to the pickup location specified bythe subsequent requesting user. In some examples, the servicearrangement system 190 (or the system 100) can transmit a message to theuser's computing device, which queries the user for an response input to(i) instruct the disposal of the object, (ii) ask the entity thatoperates the service arrangement system 190 to hold on to the object onbehalf of the user for a duration of time, (iii) request a transportservice to a specified location where the object is held, or (iv)request a delivery of the object to the user's current location (orspecified location). One or more of these options can be associated withan extra cost for the user. The service arrangement system 190 canreceive the user input, via the application on the user's computingdevice, and can process the response accordingly. For example, if theuser provides an input to receive the delivery of the object at theuser's current location, the service arrangement system 190 can processthe response as a delivery transport service request, identify theprevious AV that the object is located in, and schedule that particularAV to travel to the user's current location (e.g., before, while, orafter completing the subsequent transport service for the subsequentrequesting user). In this example, the user can be charged for thedelivery as a separate transport service in its entirety or as part ofthe user's previous transport service (e.g., as a surcharge).

While the system 100 of FIG. 1 is described as being implemented oncomputing and memory resources of an autonomous vehicle, in someexamples, the system 100 can be implemented on a mobile computing deviceoperated by a driver of a vehicle (e.g., a driver device). For example,the system 100 can be implemented as an application or program (or aspart of an application) that runs on the driver device, and thatcommunicates data with the service arrangement system 190 over one ormore networks. In such examples, the system 100 can include othercomponents, such as a user interface component that generates andpresents content (e.g., graphic user interfaces) on the display of thedriver device. When the driver is selected by the service arrangementsystem 190 to provide a transport service for a requesting user, asopposed to receiving service instructions 191, the system 100 canreceive and display an invitation to provide the transport service viathe user interface component, and the driver can either accept or rejectthe invitation. The invitation can include the user information 192, thepickup location 193, and/or the destination location 194.

Still further, in examples where the system 100 is implemented by thedriver device, the object detection 130 can receive sensor data 131 viaa communication interface of the driver device, such as a short-rangewireless communication transceiver (e.g., Bluetooth or Wi-Fi) or aphysical connector port. The driver can fix or attach, in the trunk orin the backseat of the vehicle, etc., one or more sensors which cancommunicate sensor data 131 to the object detection 130 via thecommunication interface.

As an alternative, one or more components of the system 100 can beimplemented as part of the service arrangement system 190 or as part ofanother remote system (e.g., remote from the autonomous vehicle or thedriver device) that communicates with the service arrangement system190. For example, the system 100 can transmit sensor data 131 to anobject detection 130 that is implemented in the service arrangementsystem 190, which then detects when an object has been placed in avehicle and/or has been taken out of the vehicle. The servicearrangement system 190 can then perform the remedial actions, such astransmitting notifications to the user, based on the detection.

Still further, in some examples, other sensors can be used by the system100 to detect a presence and/or removal of an object from a vehicle. Forexample, electromagnetic sensors can be used to detect electronicproducts, such as cell phones, tablets, laptop computers, etc., or otheradvanced sensors, such as chemosensors or gas chromatography sensors canbe used to detect objects using smell (e.g., food, groceries, etc.). Inone example, using electromagnetic sensors or antennas/wirelesstransceivers can be helpful to detect small electronic devices that maybe hard to detect in the vehicle, e.g., may have fallen or slippedbetween the seats or under the seat.

In a different application, the system 100 can also use the sensor datato detect whether there has been damage to the interior of the vehicleas a result of the requesting user riding in the vehicle and/ortransporting an object of the user. The system 100 can detect the stateof the vehicle's interior at a time before the transport service isinitiated (e.g., in response to when the service instructions arereceived or at a predetermined distance or estimated time away from thepickup location of the user) using sensors that are positioned in theinterior of the vehicle. At a time the transport service is beingcompleted (e.g., the vehicle is close to or has arrived at thedestination location) or has completed, the system 100 can again detectthe state of the vehicle's interior at this time, and compare theinformation with the previous information about the state of thevehicle's interior. Using this comparison, the system 100 can detectwhether something has spilled on the seats or the floor, whether theseats have been ripped or scratched, etc., after the transport service.If damage is detected, the system 100 can provide information indicatingthe damage to the service arrangement system 190 and/or to the userdevice. The service arrangement system 190 can charge the user and/orinstruct the driver or cause the vehicle to travel to a specifiedlocation for clean-up and/or repair.

Methodology

FIGS. 2 and 3 illustrate example methods for detecting objects within avehicle in connection with a transport service. Methods such asdescribed by examples of FIGS. 2 and 3 can be implemented using, forexample, components described with an example of FIG. 1. Accordingly,references made to elements of FIG. 1 are for purposes of illustrating asuitable element or component for performing a step or sub-step beingdescribed. For purpose of simplicity, the example method of FIG. 2 isdescribed as being performed by the service arrangement system 190and/or the system 100 operating on a driver device.

Referring to FIG. 2, the service arrangement system 190 can determinethat a driver of a vehicle or an AV been selected to provide a servicefor a requesting user (200). For example, the service arrangement system190 can perform a vehicle selection process to select the vehicle basedon the location of the vehicle and the service location(s) specified bythe requesting user. The service can correspond to an object or itemdelivery transport service from a pickup location to a destinationlocation, or a transport service of a person (e.g., the user) from apickup location to a destination location. The service arrangementsystem 190 can transmit, to the respective system 100 of the selectedvehicle, service instructions corresponding to the service that thedriver or the AV is to perform. In the example of an AV, the system 100can process the service instructions to control the AV to travel fromthe current location to the pickup location, while in the example of ahuman driver, the system 100 can provide route or navigation informationthat the driver can follow to move to the pickup location.

At a time after the system 100 determines that the driver or the AV isto provide the service, the system 100 can perform an object detectionoperation using data from one or more sensors of the vehicle. In oneexample, the service arrangement system 190 can receive a first set ofsensor information from the system 100 (e.g., at a first time) and candetect that an object has been placed within the vehicle based on thefirst set of information (210). The system 100 can receive the sensorinformation from a set of sensors of the vehicle. Alternatively, thesystem 100 can detect that the object has been placed within the vehicleand provide information indicating such object to the servicearrangement system 190.

Depending on implementation, one or more sensors or combination ofsensors can be used by the system 100 and/or the service arrangementsystem 190 to detect that an object has been placed in the vehicle. Forexample, the system 100 and/or the service arrangement system 190 canreceive image information from a set of cameras (or radars, LIDARs, orultrasound sensors) that are positioned within the vehicle, where theset of cameras are positioned such that the respective lens of the setof cameras faces towards an interior of the vehicle (e.g., the backseat, the front seat, the trunk, or other compartment). In anotherexample, the system 100 and/or the service arrangement system 190 canreceive weight information from a set of weighing devices that arepositioned within a compartment or the interior of the vehicle, wherethe set of weighing devices are positioned such that the respectivemeasuring platform of the set of weighing devices is positioned on,positioned below, or included with a bottom surface of the compartmentor the interior of the vehicle (e.g., the platform is at the surface ofthe trunk to weigh objects that are placed in the trunk or at or in aseat to weigh a person that sits in the back set or front seat). Stillfurther, in another example, the system 100 and/or the servicearrangement system 190 can receive compression information from a set ofsuspension sensors that is positioned proximate to or included in asuspension system of the vehicle (e.g., near the rear and/or frontsuspension system or frame of the vehicle). Additionally, a sensor canalso provide information whether a compartment or trunk has been openedand/or closed, which can be used in connection with other sensorinformation.

The system 100 and/or the service arrangement system 190 can useinformation from one or more of these different sources to perform theobject detection operation. In addition, based on the first set ofreceived sensor input/information, the system 100 and/or the servicearrangement system 190 can detect that the object has been placed in thevehicle, what the object is (or type of object), the color or size ordimension or weight of the object, what compartment the object is in,and/or the position of the object. According to an example, the servicearrangement system 190 can also associate the object information withthe trip entry of the transport service (and/or with the user's profileor account).

In various examples, the system 100 can trigger the sensors to determinesensor data (e.g., perform detection or measurement) based on locationinformation of the AV. For example, in order to conserve energy orpower, the system 100 can trigger the sensors to determine sensor datawhen the AV arrives at or close to a service location, e.g., arrives ator within a predetermined distance or estimated travel time of thepickup location, and/or when the AV is stationary for a duration of timewhile at or close to the service location. Alternatively, in anotherexample, the system 100 can trigger the sensors to determine sensor datain response to determining that the service has begun.

Referring back to FIG. 2, the service arrangement system 190 can alsodetermine that the service has begun (220). Depending on implementation,the service arrangement system 190 can determine that the service hasbegun (i) in response to detecting that the user has entered the vehicle(e.g., in response to detecting the object in the vehicle in step 210,or based on location data points of the user's computing device and thelocation data points of the vehicle, and determining that they arewithin a predetermined distance of each other), (ii) in response todetecting that a predetermined duration of time has elapsed (e.g., tenseconds) since detecting that the user has entered the vehicle, (iii) inresponse to receiving, from the user's computing device, informationindicating that the service can begin as a result of an input beingprovided at the computing device associated with the user (e.g., theuser selects a feature corresponding to starting the trip), (iv) inresponse to receiving, from driver device (or the on-board computingsystem of the AV), information indicating that the service can begin asa result of an input being provided at the driver device or the on-boardcomputing system from the driver or the user, and/or (v) in response todetermining that the vehicle has been positioned within a predetermineddistance of the start location for a predetermined duration of timebased on location information of the vehicle. In other examples, thesystem 100 detects that the service has begun based on any of theexamples (i) through (v).

The service arrangement system 190 also monitors the service, forexample, as the driver or the AV performs the service (230). Asdescribed herein, monitoring the service can include periodicallydetermining the location (and corresponding timestamp) and/or state ofthe vehicle as it travels from the pickup location to the destinationlocation, and storing such information in the trip entry associated withthe service. The system 100 can provide such information to the servicearrangement system 190. In some examples, the service arrangement system190 can also provide the information about the progress of the serviceto the user's computing device (or to another device). For example, theuser may have requested a delivery service for a package to be picked upat the user's location and dropped off at a destination location for afriend. The application on the user's device can receive (periodically)information about the progress of the service, e.g., the AV's locationand movement on a map, from the service arrangement system 190. As anaddition or an alternative, the user may have provided, to the servicearrangement system 190 via the application, the contact information(e.g., phone number or email address) of the friend who is to receivethe package. The service arrangement system 190 can monitor the serviceand provide updated information about the service to the friend, so thatthe friend can track the package as the AV travels to the destinationlocation.

The service arrangement system 190 can determine that the service hasbeen completed, e.g., as a result of monitoring the service (240).Depending on implementation, the service arrangement system 190 candetermine that the service has completed (i) in response to detectingthat the user has left the vehicle (e.g., based on location data pointsof the user's computing device and the location data points of thevehicle, and determining that they are separated by more than apredetermined distance), (ii) in response to detecting that apredetermined duration of time has elapsed (e.g., twenty seconds) sincedetecting that the user has left the vehicle, (iii) in response toreceiving, from the user's computing device, information indicating thatthe service has completed as a result of an input being provided at thecomputing device associated with the user (e.g., the user selects afeature corresponding to ending the trip), (iv) in response toreceiving, from driver device (or the on-board computing system of theAV), information indicating that the service has completed as a resultof an input being provided at the driver device or the on-boardcomputing system from the driver or the user, and/or (v) in response todetermining that the vehicle has been positioned within a predetermineddistance of the destination location for a predetermined duration oftime based on location information of the vehicle.

After determining that the service has been completed, the servicearrangement system 190 can detect whether the object (that waspreviously detected in the vehicle) has been removed from the vehiclebased on a second set of sensor information from one or more sensors ofthe vehicle (e.g., at a second time) (250). For example, the servicearrangement system 190 can receive the second set of sensor informationfrom the system 100 and can compare the first and second sets of sensorinformation to determine if there is a difference, e.g., difference inimagery, difference in weight, difference in compression information,etc. As an alternative, the system 100 can perform the comparisonoperation and provide an indication of whether the object has beenremoved to the service arrangement system 190. The system 100 cantrigger the sensors to determine sensor data (e.g., perform detection ormeasurement) at a particular instance in time, depending on examples,such as in response to detecting that the service has been completed, orafter a predetermined duration of time since detecting that the servicehas been completed.

Alternatively, the service arrangement system 190 can determine whetherthe object has been removed from the vehicle before the service isdetermined to be completed. For example, the service arrangement system190 (or the system 100) can perform the object detection process inresponse to determining that the AV has reached or is close to thedestination location and/or has been stationary for a duration of timewhile at or close to the destination location. In another example, theservice arrangement system 190 can determine that the service has beencompleted in response to detecting that the object has been removed fromthe vehicle (e.g., after performing step 250).

If the service arrangement system 190 determines that the object hasbeen removed from the vehicle, it can determine that no remedial actionneeds to be performed, as the user in the vehicle has taken the objectwith him or her, or another user has removed the object (e.g., at thedelivery destination location) (270). On the other hand, if the servicearrangement system 190 determines that the object has not been removed,the service arrangement system 190 can automatically perform one or moreremedial actions (260). For example, the service arrangement system 190can message the driver and/or the user, via the respective applicationson the driver device and/or the user device, to inform them about themisplaced object. The message can include instructions (e.g., includingnavigation instructions) for enabling the driver and the user to meet upat a location, e.g., the destination location, to enable the user toretrieve the object.

In another example, the service arrangement system 190 can transmit amessage or notification to the user device and enable the user torequest a delivery of the object to a location specified by the user.Still further, in one example, if the vehicle is an AV, the servicearrangement system 190 can transmit vehicle control instructions to theAV to cause the AV to travel to a specified location. Such a specifiedlocation can correspond to a new meeting location (delivery drop offlocation) for the user specified in the user's subsequent deliverytransport service request, or a designated storage or warehouse locationto house misplaced objects. The service arrangement system 190 can alsomessage the user device to provide such location information on wherethe user can retrieve the object.

FIG. 3 illustrates another example method for detecting objects within avehicle in connection with a transport service. In one example, one ormore steps of the method described in FIG. 3 can be performedconcurrently with one or more steps of the method described in FIG. 2.For purpose of simplicity, the example method of FIG. 3 is described asbeing performed by the system 100 operating on an on-board computingsystem of an autonomous vehicle.

In FIG. 3, the system 100 (of the AV) can receive service instructionsfrom the service arrangement system 190 for the AV to provide a servicefor a requesting user (300). As described herein, a service can have astart location and a destination location, and can correspond to adelivery of an object or a person (e.g., the requesting user). Thesystem 100 can process the service instructions in order to determinewhat service to provide, where to travel to (e.g., from the currentlocation, to the start/pickup location, then to the destinationlocation), and how to travel there (e.g., the route to take).

Based on the service instructions, map data, and/or sensor data used foroperating the AV (e.g., data from sensors mounted on the body of theAV), the system 100 can control the AV to travel to the start location,and concurrently, monitor the movement of the AV (310). The system 100can correspond to, include, or be in communication with the AV controlsystem to control the operation of the AV by sending control signals tovehicle control modules of various sub-systems in response to receivingsensor data from the sensors and feedback signals from various vehiclecomponents. The system 100 can also determine the location of the AVfrom GPS data, for example, and monitor the current position as comparedto the route and/or the start location. Using the periodicallydetermined current location of the AV, the system 100 can determine whenthe AV arrives at or is within a predetermined distance of the startlocation (e.g., the AV is close to the start location). In one example,when the system 100 determines that the AV is close to the startlocation, the system 100 can change the state of the AV (from the onroute state to the arriving now or has arrived state).

At a time after determining that the AV is close to the start location,the system 100 can perform an object detection process to detect that anobject has been placed within the AV (320). Depending on implementation,the system 100 can periodically perform the object detection process, asdescribed in FIGS. 1 and 2, from the time after determining that the AVis close to the start location to a time after the AV begins travelingto the destination location. In one example, the system 100 can performthe object detection process in response to determining that the servicehas started or can perform the object detection process before orconcurrently while determining that the service has started. In theexample of FIG. 3, the system 100 can detect that that an object (e.g.,a box) has been placed in the back seat of the AV from sensorinformation received from the AV, as described in FIGS. 1 and 2.

Once the service has started, the system 100 can control the AV totravel to the destination location, and also monitor the vehicle and/orthe progress of the service (330). Similarly as described above, thesystem 100 can determine the location of the AV from GPS data andmonitor the current position as compared to the route and/or thedestination location. The system 100 can determine when the AV arrivesat or is within a predetermined distance of the destination location(e.g., the AV is close to the destination location) (340). At a timeafter determining that the AV is close to the destination location, thesystem 100 can perform an object detection process to detect whether theobject has been removed from within the AV (350). Depending onimplementation, step 350 can be performed before, after, or whiledetermining that the service has been completed.

If the system 100 determines that the object has been removed from theAV, the system 100 can determine that no remedial action needs to beperformed (370). On the other hand, if the system 100 determines thatthe object is still present in the vehicle, the system 100 can performone or more remedial actions and/or cause the service arrangement system190 to perform one or more remedial actions, such as described in FIGS.1 and 2 (360).

FIGS. 4A through 4D illustrate examples of user interfaces displayed ona mobile computing device in various use case examples. For each of theexamples illustrated in FIGS. 4A through 4D, an individual userinterface can be displayed on a user device of a user who has left anobject in a vehicle in connection with the user's transport service.Depending on implementation, the user interface can be displayed as partof a designated application associated with the service arrangementsystem 190 or as part of another application operating on the userdevice (e.g., email application, text message application, a web browserapplication, etc.). In one example, a notification can be displayed onthe user device, and selection of a feature or notification can causethe user interface to be displayed on the user device.

Referring to FIG. 4A, in one example, the user device can receive anotification or a message from the service arrangement system 190 or thesystem 100 after (or in some instances, just before) the service hasended. The application running on the user device can display the userinterface 400, which includes image content 405 of the object andtextual content 410 reminding the user and/or notifying the user thatthe object was left in the vehicle. The image content 405 can begenerated from image information from one or more sensors (e.g.,cameras) of the vehicle. Alternatively, if a camera is not used in thevehicle, other content representing the object can be displayed in placeof the image content 405, such as additional textual content or othergraphics or placeholder images representing the object. The textualcontent 410, in this example, also provides a contact information forthe user to message in order to determine what to do in order toretrieve the object

In the example of FIG. 4B, a user interface 420 can be displayed on theuser device. The user interface 420 can also include image content 425,such as described in FIG. 4A, and textual content 430 notifying the userthat the object was left in the vehicle and informing the user of aspecific location in which to pick up the object. For example, theservice arrangement system 190 can instruct the AV or the driver totravel to a specific location (or remain at the destination location orwhere the user was dropped off). The service arrangement system 190 cangenerate textual content 430 to include corresponding content about thelocation. In FIG. 4B, the textual content 430 informs the user that thevehicle with the object will be waiting for 2 more minutes (or someother predetermined duration of time) where the user was dropped off.

According to another example, a user interface 440 of FIG. 4C can bedisplayed on the user device as an interactive notification prompt. Sucha notification prompt can be in-app (e.g., displayed as part of theapplication user interface) or out-of-app (e.g., displayed when theapplication is not currently open and using native user interfacenotification features of an operating system of a mobile device). Inthis example, the user interface 440 includes textual content 445 thatincludes a specific location at which the user's object will be storedor held. The textual content 445 can include location information (e.g.,address or street intersection), a time or duration of time when theuser can pick up the object (e.g., “normal business hours” or “8 am to 6pm”), and a selectable feature 450 that the user can select in order torequest a transport service to the specified location. If the userselects the selectable feature (“Request” or “Request trip”) 450, thedesignated service application can be opened or launched on the userdevice, so that the user can confirm the transport service to thespecified location (e.g., from the user's current location or from aspecified pickup location). Alternatively, if the user selects theselectable feature 450, the designated service application canautomatically generate and transmit a request for the transport serviceto the service arrangement system 190. The request can include the typeof service (e.g., transport the user), the pickup location (e.g., thecurrent location of the user device determined via the GPS receiver ofthe user device), and the destination location (e.g., 123 Main Street).

In another example, such as illustrated in FIG. 4D, a user interface 460can include textual content 465 to inform the user that the user canrequest a delivery of the user's object. The user interface 460 canenable the user to select a selectable feature 470 to request a deliverytransport service of the user's object to the user's current location.Similar to the example of FIG. 4C, if the user selects the selectablefeature 470, the designated service application can be opened orlaunched on the user device, so that the user can confirm the deliverytransport service of the user's object to the specified location (e.g.,from the user's current location or from a specified pickup location) orthe designated service application can automatically generate andtransmit a request for the delivery transport service to the servicearrangement system 190. In the latter example, the request can includethe type of service (e.g., delivery of the object in the specificvehicle that has the user's object), and the drop off or destinationlocation (e.g., the current location of the user device determined viathe GPS receiver of the user device). As described in FIG. 1, forexample, the service arrangement system 190 can identify the particularvehicle that the user had just previously used or the vehicle that isassociated with the trip entry having the identified object, and selectthat vehicle to go to the user's current location.

According to some examples, a user interface described in any of FIGS.4A through 4D can also include detailed information about the object,such as what the object is (including color or shape), the object type,the dimensions and/or weight of the object, where the object was placed,etc. While the examples of FIGS. 4A through 4D are described withrespect to the user device, in some variations, such user interfaces canalso be displayed on the driver device to inform the driver of an objectthat has been left behind in his or her vehicle. Such a user interfacecan instruct the driver where to go, where to wait (and/or for howlong), and/or who to meet with, so that the user can retrieve theobject.

Hardware Diagrams

FIG. 5 is a block diagram that illustrates a computing device upon whichembodiments described herein may be implemented. In one example, acomputing device 500 may correspond to a mobile computing device, suchas a cellular device that is capable of telephony, messaging, and dataservices. The computing device 500 can correspond to a user device or adriver device. Examples of such devices include smartphones, handsets ortablet devices for cellular carriers. The computing device 500 includesa processor 510, memory resources 520, a display device 530 (e.g., suchas a touch-sensitive display device), one or more communicationsub-systems 540 (including wireless communication sub-systems), inputmechanisms 550 (e.g., an input mechanism can include or be part of thetouch-sensitive display device), and one or more sensors 560, includinga location detection mechanism (e.g., GPS receiver). In one example, atleast one of the communication sub-systems 540 sends and receivescellular data over data channels and voice channels. The communicationssub-systems 540 can include a cellular transceiver and one or moreshort-range wireless transceivers. The processing resources 510 canreceive communications from the service arrangement system, for example,via the communications sub-systems 540.

In the example of FIG. 5, the computing device 500 can correspond to adriver device that is operated by a driver of a vehicle. The processor510 can provide a variety of content to the display 530 by executinginstructions stored in the memory resources 520. The memory resources520 can store instructions corresponding to the driver application 525,for example, and other data, such as one or more trip entries 527, oneor more parameters, or databases of location data points. For example,the processor 510 is configured with software and/or other logic toperform one or more processes, steps, and other functions described withimplementations, such as described by FIGS. 1 through 4D, and elsewherein the application. In particular, the processor 510 can executeinstructions and data stored in the memory resources 520 in order to benotified about a transport service the driver is to provide, and inconnection with the transport service, can use sensor data 565 toperform object detection processes. The sensors 560, as representedherein, can include a sensor included with the computing device 500(e.g., a GPS receiver) as well as sensor interfaces that communicatewith one or more sensors of the vehicle (e.g., ultrasound sensors,weighing devices, etc.). The processor 510 can also execute instructionscorresponding to the driver application 525 to cause various userinterfaces 515 to be displayed on the display 530, such as an invitationto provide a transport service, navigation information, or notificationsabout a user having left an object in the driver's vehicle. In addition,the user interfaces 515 can include selectable features that the drivercan select, by providing user input 555 via the input mechanisms 550(e.g., in combination with a touch-sensitive display 530), to interactwith the driver application 525, such as to indicate when the transportservice has started or completed, in some examples.

While FIG. 5 is illustrated for a mobile computing device, one or moreexamples may be implemented on other types of devices, includingfull-functional computers, such as laptops and desktops (e.g., PC). Inanother example, a variation of FIG. 5 can illustrate (at least aportion of) an on-board computing system of an AV, such has described inFIG. 1.

FIG. 6 is a block diagram that illustrates a computer system upon whichexamples described herein may be implemented. For example, in thecontext of FIG. 1, the service arrangement system 190 may be implementedusing a computer system such as described by FIG. 6. The servicearrangement system 190 may also be implemented using a combination ofmultiple computer systems as described by FIG. 6.

In one implementation, the computer system 600 includes processingresources, such as one or more processors 610, a main memory 620, aread-only memory (ROM) 630, a storage device 640, and a communicationinterface 650. The computer system 600 includes at least one processor610 for processing information and the main memory 620, such as a randomaccess memory (RAM) or other dynamic storage device, for storinginformation and instructions to be executed by the processor 610. Themain memory 620 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by the processor 610. The computer system 600 may also includethe ROM 630 or other static storage device for storing staticinformation and instructions for the processor 610. The storage device640, such as a magnetic disk or optical disk, is provided for storinginformation and instructions. For example, the storage device 640 cancorrespond to a computer-readable medium that stores locationdetermination instructions 642 and object detection instructions 644 forperforming operations discussed with respect to FIGS. 1 through 5. Insuch examples, the computer system 600 can determine the locations ofthe vehicles, determine the service locations of the transport services,and perform object detection processes based on sensor data receivedfrom the vehicle systems or driver devices. In addition, the storagedevice 640 can include other instructions, such as instructions toimplement a dispatch sub-system, which receives requests for transportservices from requesting users and selects vehicles to provide thosetransport services.

The communication interface 650 can enable the computer system 600 tocommunicate with one or more networks 680 (e.g., cellular network)through use of the network link (wirelessly or using a wire). Using thenetwork link, the computer system 600 can communicate with a pluralityof devices, such as the mobile computing devices of the riders anddrivers. According to some examples, using the communication interface650, the computer system 600 can receive location information 652 fromthe driver devices, AVs, and/or user devices, and can transmitnotification messages 654 to user devices to inform users if users'belongings have been left in a vehicle, such as described by someexamples of FIGS. 1 through 5.

The computer system 600 can also include a display device 660, such as acathode ray tube (CRT), an LCD monitor, or a television set, forexample, for displaying graphics and information to a user. An inputmechanism 670, such as a keyboard that includes alphanumeric keys andother keys, can be coupled to the computer system 600 for communicatinginformation and command selections to the processor 610. Othernon-limiting, illustrative examples of the input mechanisms 670 includea mouse, a trackball, touch-sensitive screen, or cursor direction keysfor communicating direction information and command selections to theprocessor 610 and for controlling cursor movement on the display 660.

Examples described herein are related to the use of the computer system600 for implementing the techniques described herein. According to oneexample, those techniques are performed by the computer system 600 inresponse to the processor 610 executing one or more sequences of one ormore instructions contained in the main memory 620. Such instructionsmay be read into the main memory 620 from another machine-readablemedium, such as the storage device 640. Execution of the sequences ofinstructions contained in the main memory 620 causes the processor 610to perform the process steps described herein. In alternativeimplementations, hard-wired circuitry may be used in place of or incombination with software instructions to implement examples describedherein. Thus, the examples described are not limited to any specificcombination of hardware circuitry and software.

It is contemplated for examples described herein to extend to individualelements and concepts described herein, independently of other concepts,ideas or system, as well as for examples to include combinations ofelements recited anywhere in this application. Although examples aredescribed in detail herein with reference to the accompanying drawings,it is to be understood that the concepts are not limited to thoseprecise examples. Accordingly, it is intended that the scope of theconcepts be defined by the following claims and their equivalents.Furthermore, it is contemplated that a particular feature describedeither individually or as part of an example can be combined with otherindividually described features, or parts of other examples, even if theother features and examples make no mentioned of the particular feature.Thus, the absence of describing combinations should not preclude havingrights to such combinations.

What is being claimed is:
 1. A method for monitoring a vehicle that isoperating to provide transport services, the method being implemented byone or more processors and comprising: receiving, from a remote system,instructions for the vehicle to provide a first transport service from astart location to a destination; receiving first image information of aninterior of the vehicle from one or more cameras positioned within thevehicle; identifying, from the first image information, an object thathas been placed in the interior of the vehicle in connection with thefirst transport service; prior to completion of the first transportservice, receiving second image information of the interior of thevehicle from the one or more cameras positioned within the vehicle;determining that the object has been removed from the interior of thevehicle by comparing the first image information and the second imageinformation; and in response to determining that the object has beenremoved from the interior of the vehicle, initiating performance of aremedial action.
 2. The method of claim 1, further comprising performingan image recognition process on the first image information and thesecond image information to recognize the object based on visualcharacteristics.
 3. The method of claim 1, wherein the vehicle receivesthe second image information and compares the first image informationand the second image information as the vehicle travels to thedestination.
 4. The method of claim 1, wherein the first transportservice is a delivery service and the object is an item to be delivered.5. The method of claim 1, wherein initiating performance of the remedialaction includes automatically transmitting a notification that theobject is no longer present in the vehicle to a computing system thatarranged the first transport service.
 6. The method of claim 1, whereininitiating performance of the remedial action includes automaticallycommunicating, over one or more networks, a message to a computingdevice associated with a driver of the vehicle, the message notifyingthe vehicle that the object has been removed from the interior of thevehicle.
 7. The method of claim 6, wherein the message includes an imageof the object or results of the comparison between the first imageinformation and the second image information.
 8. The method of claim 1,further comprising: determining that the object was placed in thevehicle by a user who entered the vehicle at the start location.
 9. Themethod of claim 8, wherein initiating performance of the remedial actionincludes automatically communicating, over one or more networks, amessage to a computing device associated with the user, the messagenotifying the user that the object has been removed from the interior ofthe vehicle.
 10. The method of claim 8, further comprising: associatingthe object with a profile or account of the user.
 11. A vehiclecomprising: one or more sensors generating sensor data; and a controlsystem to execute instructions that cause the vehicle to performoperations including: receiving, from a remote system, instructions forthe vehicle to provide a first transport service from a start locationto a destination; receiving first image information of an interior ofthe vehicle from one or more cameras positioned within the vehicle;identifying, from the first image information, an object that has beenplaced in the interior of the vehicle in connection with the firsttransport service; prior to completion of the first transport service,receiving second image information of the interior of the vehicle fromthe one or more cameras positioned within the vehicle; determining thatthe object has been removed from the interior of the vehicle bycomparing the first image information and the second image information;and in response to determining that the object has been removed from theinterior of the vehicle, initiating performance of a remedial action.12. The vehicle of claim 11, further comprising performing an imagerecognition process on the first image information and the second imageinformation to recognize the object based on visual characteristics. 13.The vehicle of claim 11, wherein the vehicle receives the second imageinformation and compares the first image information and the secondimage information as the vehicle travels to the destination.
 14. Thevehicle of claim 11, wherein the first transport service is a deliveryservice and the object is an item to be delivered.
 15. The vehicle ofclaim 11, wherein initiating performance of the remedial action includesautomatically transmitting a notification that the object is no longerpresent in the vehicle to a computing system that arranged the firsttransport service.
 16. The vehicle of claim 11, wherein initiatingperformance of the remedial action includes automatically communicating,over one or more networks, a message to a computing device associatedwith a driver of the vehicle, the message notifying the vehicle that theobject has been removed from the interior of the vehicle.
 17. Thevehicle of claim 16, wherein the message includes an image of the objector results of the comparison between the first image information and thesecond image information.
 18. The vehicle of claim 11, furthercomprising: determining that the object was placed in the vehicle by auser who entered the vehicle at the start location.
 19. The vehicle ofclaim 18, wherein initiating performance of the remedial action includesautomatically communicating, over one or more networks, a message to acomputing device associated with the user, the message notifying theuser that the object has been removed from the interior of the vehicle.20. A non-transitory computer-readable medium storing instructions that,when executed by one or more processors of an on-board computing systemof a vehicle, cause the on-board computing system to perform operationsincluding: receiving, from a remote system, instructions for the vehicleto provide a first transport service from a start location to adestination; receiving first image information of an interior of thevehicle from one or more cameras positioned within the vehicle;identifying, from the first image information, an object that has beenplaced in the interior of the vehicle in connection with the firsttransport service; prior to completion of the first transport service,receiving second image information of the interior of the vehicle fromthe one or more cameras positioned within the vehicle; determining thatthe object has been removed from the interior of the vehicle bycomparing the first image information and the second image information;and in response to determining that the object has been removed from theinterior of the vehicle, initiating performance of a remedial action.